Eye tracking system with single point calibration

ABSTRACT

A head mounted display (HMD) comprises an eye tracking system configured to perform a calibration process using an eye tracking system of the HMD that includes determining a pupillary axis and/or determining an angular offset between the pupillary axis and the eye&#39;s true line of sight. The eye tracking system obtains an eye model captures images of the user&#39;s pupil while the user is looking at a target or other content displayed on the HMD. In some embodiments, the calibration process is based on a single image of the user&#39;s eye and is performed only once. For example, the process can be performed the first time the user uses the HMD, which stores the calibration data for the user in a memory for future use.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/456,383, filed Mar. 10, 2017, which claims the benefit of U.S.Application No. 62/306,777, filed Mar. 11, 2016 which is incorporated byreference in its entirety.

BACKGROUND

The present disclosure generally relates to eye tracking, andspecifically to an eye tracking system with a single point calibration.

Eye tracking is an important feature for head-mounted display (HMD)systems including systems used in virtual reality (VR) applications.Calibration of such tracking systems is important for accuracy of thedata reported by HMD systems. Conventionally, calibration of HMD systemsinvolved a significant calibration process and typically requiredcalibration of the tracking system before every use by the user, whichreduces user experience. For example, conventional calibration processestypically ask the user to look at a sequence of targets before everyuse. A calibration system that improves user experience by eitherreducing the number of targets the user has to look or reducing thefrequency of calibration is useful.

SUMMARY

A calibration process for an eye tracking system includes determining apupillary axis and/or determining an angular offset between thepupillary axis and the eye's true line of sight for each eye. The eyetracking system obtains an eye model (e.g., either generates itself orreceives from external to the eye tracking system) for the eye thatincludes information of radius and origin of a corneal sphere for eacheye, where each eye is approximated as two spheres with one sphereapproximating the overall eye and a portion of the other sphere (i.e.,corneal sphere) approximating a portion of the cornea. The calibrationprocess includes capturing images of the user's pupil while the user islooking at one or more known targets (e.g., specific points on anelectronic display of the VR headset) or while viewing content displayedon the HMD in a normal mode of operation. In one embodiment, a singlepoint calibration is performed where the user only needs to look at onespecific target on the electronic display.

The eye tracking system identifies a shape of the pupil in each of thecaptured images of the user's eye and then uses the identified shape ofthe pupil along with the obtained eye model information to determine a3D plane in which the pupil is located. An observable axis is thenderived that identifies a ray originating from the 3D plane and isnormal to the corneal surface of the corneal sphere. This axis isdefined herein as the pupillary axis of the eye. The system may alsodetermine an angular offset between the pupillary axis and the eye'strue line of sight (i.e., represented by location of fovea). The systemmay further determine torsional state of the user's eye or determineuser identification by tracking the user's iris along with thedetermined pupillary axis and/or the angular offset. In someembodiments, the calibration process for determining the pupillary axisand/or the angular offset between the pupillary axis and the user's trueline of sight is based on a single image of the user's eye and isperformed only once. For example, the calibration process is performedthe very first time the user puts the VR headset on and the VR headsetstores the calibration data corresponding to the user in a memory foruse in future uses by the same user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment including a VR system,in accordance with an embodiment.

FIG. 2A is a diagram of a VR headset, in accordance with an embodiment.

FIG. 2B is a cross section of a front rigid body of the VR headset inFIG. 2A, in accordance with an embodiment.

FIG. 3 depicts an example eye tracking system for generating an eyemodel, in accordance with an embodiment.

FIG. 4 is a flowchart of an example process for generating an eye modelusing an eye tracking system, in accordance with an embodiment.

FIG. 5 depicts an example one-time single point calibration for an HMDsystem, in accordance with an embodiment.

FIG. 6 is a flowchart of an example process for a one-time single pointcalibration of an HMD system, in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

System Overview

FIG. 1 is a block diagram of a VR system environment 100 in which a VRconsole 110 operates. The system environment 100 shown by FIG. 1comprises a VR headset 105, an imaging device 135, and a VR inputinterface 140 that are each coupled to the VR console 110. While FIG. 1shows an example system 100 including one VR headset 105, one imagingdevice 135, and one VR input interface 140, in other embodiments anynumber of these components may be included in the system 100. Forexample, there may be multiple VR headsets 105 each having an associatedVR input interface 140 and being monitored by one or more imagingdevices 135, with each VR headset 105, VR input interface 140, andimaging devices 135 communicating with the VR console 110. Inalternative configurations, different and/or additional components maybe included in the system environment 100.

The VR headset 105 is a HMD that presents content to a user. Examples ofcontent presented by the VR head set include one or more images, video,audio, or some combination thereof. In some embodiments, audio ispresented via an external device (e.g., speakers and/or headphones) thatreceives audio information from the VR headset 105, the VR console 110,or both, and presents audio data based on the audio information. Anembodiment of the VR headset 105 is further described below inconjunction with FIGS. 2A and 2B. The VR headset 105 may comprise one ormore rigid bodies, which may be rigidly or non-rigidly coupled to eachother together. A rigid coupling between rigid bodies causes the coupledrigid bodies to act as a single rigid entity. In contrast, a non-rigidcoupling between rigid bodies allows the rigid bodies to move relativeto each other.

The VR headset 105 includes an electronic display 115, an optics block118, one or more locators 120, one or more position sensors 125, aninertial measurement unit (IMU) 130, and an eye tracking system 160. Theelectronic display 115 displays images to the user in accordance withdata received from the VR console 110. In various embodiments, theelectronic display 115 may comprise a single electronic display ormultiple electronic displays (e.g., a display for each eye of a user).Examples of the electronic display 115 include: a liquid crystal display(LCD), an organic light emitting diode (OLED) display, an active-matrixorganic light-emitting diode display (AMOLED), some other display, orsome combination thereof.

The optics block 118 magnifies received light from the electronicdisplay 115, corrects optical errors associated with the image light,and the corrected image light is presented to a user of the VR headset105. An optical element may be an aperture, a Fresnel lens, a convexlens, a concave lens, a filter, or any other suitable optical elementthat affects the image light emitted from the electronic display 115.Moreover, the optics block 118 may include combinations of differentoptical elements. In some embodiments, one or more of the opticalelements in the optics block 118 may have one or more coatings, such asanti-reflective coatings.

Magnification of the image light by the optics block 118 allows theelectronic display 115 to be physically smaller, weigh less, and consumeless power than larger displays. Additionally, magnification mayincrease a field of view of the displayed content. For example, thefield of view of the displayed content is such that the displayedcontent is presented using almost all (e.g., 110 degrees diagonal), andin some cases all, of the user's field of view. In some embodiments, theoptics block 118 is designed so its effective focal length is largerthan the spacing to the electronic display 115, which magnifies theimage light projected by the electronic display 115. Additionally, insome embodiments, the amount of magnification may be adjusted by addingor removing optical elements.

The optics block 118 may be designed to correct one or more types ofoptical errors in addition to fixed pattern noise (i.e., the screen dooreffect). Examples of optical errors include: two-dimensional opticalerrors, three-dimensional optical errors, or some combination thereof.Two-dimensional errors are optical aberrations that occur in twodimensions. Example types of two-dimensional errors include: barreldistortion, pincushion distortion, longitudinal chromatic aberration,transverse chromatic aberration, or any other type of two-dimensionaloptical error. Three-dimensional errors are optical errors that occur inthree dimensions. Example types of three-dimensional errors includespherical aberration, comatic aberration, field curvature, astigmatism,or any other type of three-dimensional optical error. In someembodiments, content provided to the electronic display 115 for displayis pre-distorted, and the optics block 118 corrects the distortion whenit receives image light from the electronic display 115 generated basedon the content.

The locators 120 are objects located in specific positions on the VRheadset 105 relative to one another and relative to a specific referencepoint on the VR headset 105. A locator 120 may be a light emitting diode(LED), a corner cube reflector, a reflective marker, a type of lightsource that contrasts with an environment in which the VR headset 105operates, or some combination thereof. In embodiments where the locators120 are active (i.e., an LED or other type of light emitting device),the locators 120 may emit light in the visible band (−380 nm to 750 nm),in the infrared (IR) band (−750 nm to 1 mm), in the ultraviolet band (10nm to 380 nm), some other portion of the electromagnetic spectrum, orsome combination thereof.

In some embodiments, the locators 120 are located beneath an outersurface of the VR headset 105, which is transparent to the wavelengthsof light emitted or reflected by the locators 120 or is thin enough notto substantially attenuate the wavelengths of light emitted or reflectedby the locators 120. Additionally, in some embodiments, the outersurface or other portions of the VR headset 105 are opaque in thevisible band of wavelengths of light. Thus, the locators 120 may emitlight in the IR band under an outer surface that is transparent in theIR band but opaque in the visible band.

The IMU 130 is an electronic device that generates fast calibration databased on measurement signals received from one or more of the positionsensors 125. A position sensor 125 generates one or more measurementsignals in response to motion of the VR headset 105. Examples ofposition sensors 125 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, another suitable type of sensorthat detects motion, a type of sensor used for error correction of theIMU 130, or some combination thereof. The position sensors 125 may belocated external to the IMU 130, internal to the IMU 130, or somecombination thereof.

Based on the one or more measurement signals from one or more positionsensors 125, the IMU 130 generates fast calibration data indicating anestimated position of the VR headset 105 relative to an initial positionof the VR headset 105. For example, the position sensors 125 includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, roll). In some embodiments, the IMU 130rapidly samples the measurement signals and calculates the estimatedposition of the VR headset 105 from the sampled data. For example, theIMU 130 integrates the measurement signals received from theaccelerometers over time to estimate a velocity vector and integratesthe velocity vector over time to determine an estimated position of areference point on the VR headset 105. Alternatively, the IMU 130provides the sampled measurement signals to the VR console 110, whichdetermines the fast calibration data. The reference point is a pointthat may be used to describe the position of the VR headset 105. Whilethe reference point may generally be defined as a point in space;however, in practice the reference point is defined as a point withinthe VR headset 105 (e.g., a center of the IMU 130).

The IMU 130 receives one or more calibration parameters from the VRconsole 110. As further discussed below, the one or more calibrationparameters are used to maintain tracking of the VR headset 105. Based ona received calibration parameter, the IMU 130 may adjust one or more IMUparameters (e.g., sample rate). In some embodiments, certain calibrationparameters cause the IMU 130 to update an initial position of thereference point so it corresponds to a next calibrated position of thereference point. Updating the initial position of the reference point asthe next calibrated position of the reference point helps reduceaccumulated error associated with the determined estimated position. Theaccumulated error, also referred to as drift error, causes the estimatedposition of the reference point to “drift” away from the actual positionof the reference point over time.

The eye tracking system 160 generates the eye model using an exampleprocess described in conjunction with FIG. 4. The eye tracking system160 includes an eye tracking unit and a control module. The eye trackingunit is located within an HMD system (e.g., VR system 100 or othersystems such as an AR system) and includes, among other components,illumination sources and optical sensors. The illumination sources(e.g., point light sources) and optical sensors (e.g., camera) of theeye tracking unit are used for corneal sphere tracking to determine amodel of an eye of a user while the user is wearing the HMD (e.g., VRheadset 105). The control module may also perform a calibration processfor the HMD system to determine a pupillary axis and/or an angularoffset between the pupillary axis and the user's true line of sight. Theillumination sources and the optical sensors are coupled to the controlmodule that performs the necessary data processing for generating theeye model, performing the calibration of the HMD system, and performingoptical actions. The control module is located within the VR headset 105and/or the VR console 110.

The imaging device 135 generates slow calibration data in accordancewith calibration parameters received from the VR console 110. Slowcalibration data includes one or more images showing observed positionsof the locators 120 that are detectable by the imaging device 135. Theimaging device 135 may include one or more cameras, one or more videocameras, any other device capable of capturing images including one ormore of the locators 120, or some combination thereof. Additionally, theimaging device 135 may include one or more filters (e.g., used toincrease signal to noise ratio). The imaging device 135 is configured todetect light emitted or reflected from locators 120 in a field of viewof the imaging device 135. In embodiments where the locators 120 includepassive elements (e.g., a retroreflector), the imaging device 135 mayinclude a light source that illuminates some or all of the locators 120,which retro-reflect the light towards the light source in the imagingdevice 135. Slow calibration data is communicated from the imagingdevice 135 to the VR console 110, and the imaging device 135 receivesone or more calibration parameters from the VR console 110 to adjust oneor more imaging parameters (e.g., focal length, focus, frame rate, ISO,sensor temperature, shutter speed, aperture, etc.).

The VR input interface 140 is a device that allows a user to send actionrequests to the VR console 110. An action request is a request toperform a particular action. For example, an action request may be tostart or end an application or to perform a particular action within theapplication. The VR input interface 140 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the received action requests to the VR console 110. Anaction request received by the VR input interface 140 is communicated tothe VR console 110, which performs an action corresponding to the actionrequest. In some embodiments, the VR input interface 140 may providehaptic feedback to the user in accordance with instructions receivedfrom the VR console 110. For example, haptic feedback is provided whenan action request is received, or the VR console 110 communicatesinstructions to the VR input interface 140 causing the VR inputinterface 140 to generate haptic feedback when the VR console 110performs an action.

The VR console 110 provides content to the VR headset 105 forpresentation to the user in accordance with information received fromone or more of: the imaging device 135, the VR headset 105, and the VRinput interface 140. In the example shown in FIG. 1, the VR console 110includes an application store 145, a tracking module 150, and a VRengine 155. Some embodiments of the VR console 110 have differentmodules than those described in conjunction with FIG. 1. Similarly, thefunctions further described below may be distributed among components ofthe VR console 110 in a different manner than is described here.

The application store 145 stores one or more applications for executionby the VR console 110. An application is a group of instructions, thatwhen executed by a processor, generates content for presentation to theuser. Content generated by an application may be in response to inputsreceived from the user via movement of the VR headset 105 or the VRinterface device 140. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

The tracking module 150 calibrates the VR system 100 using one or morecalibration parameters and may adjust one or more calibration parametersto reduce error in determination of the position of the VR headset 105.For example, the tracking module 150 adjusts the focus of the imagingdevice 135 to obtain a more accurate position for observed locators onthe VR headset 105. Moreover, calibration performed by the trackingmodule 150 also accounts for information received from the IMU 130.Additionally, if tracking of the VR headset 105 is lost (e.g., theimaging device 135 loses line of sight of at least a threshold number ofthe locators 120), the tracking module 140 re-calibrates some or theentire system environment 100.

The tracking module 150 tracks movements of the VR headset 105 usingslow calibration information from the imaging device 135. The trackingmodule 150 determines positions of a reference point of the VR headset105 using observed locators from the slow calibration information and amodel of the VR headset 105. The tracking module 150 also determinespositions of a reference point of the VR headset 105 using positioninformation from the fast calibration information. Additionally, in someembodiments, the tracking module 150 may use portions of the fastcalibration information, the slow calibration information, or somecombination thereof, to predict a future location of the headset 105.The tracking module 150 provides the estimated or predicted futureposition of the VR headset 105 to the VR engine 155.

The VR engine 155 executes applications within the system environment100 and receives position information, acceleration information,velocity information, predicted future positions, or some combinationthereof of the VR headset 105 from the tracking module 150. Based on thereceived information, the VR engine 155 determines content to provide tothe VR headset 105 for presentation to the user. For example, if thereceived information indicates that the user has looked to the left, theVR engine 155 generates content for the VR headset 105 that mirrors theuser's movement in a virtual environment. Additionally, the VR engine155 performs an action within an application executing on the VR console110 in response to an action request received from the VR inputinterface 140 and provides feedback to the user that the action wasperformed. The provided feedback may be visual or audible feedback viathe VR headset 105 or haptic feedback via the VR input interface 140.

FIG. 2A is a diagram of a VR headset, in accordance with an embodiment.The VR headset 200 is an embodiment of the VR headset 105, and includesa front rigid body 205 and a band 210. The front rigid body 205 includesthe electronic display 115 (not shown in FIG. 2A), the IMU 130, the oneor more position sensors 125, and the locators 120. In the embodimentshown by FIG. 2A, the position sensors 125 are located within the IMU130, and neither the IMU 130 nor the position sensors 125 are visible tothe user.

The locators 120 are located in fixed positions on the front rigid body205 relative to one another and relative to a reference point 215. Inthe example of FIG. 2A, the reference point 215 is located at the centerof the IMU 130. Each of the locators 120 emit light that is detectableby the imaging device 135. Locators 120, or portions of locators 120,are located on a front side 220A, a top side 220B, a bottom side 220C, aright side 220D, and a left side 220E of the front rigid body 205 in theexample of FIG. 2A.

FIG. 2B is a cross section 225 of the front rigid body 205 of theembodiment of a VR headset 200 shown in FIG. 2A. As shown in FIG. 2B,the front rigid body 205 includes an optical block 230 that providesaltered image light to an exit pupil 250. The exit pupil 250 is thelocation of the front rigid body 205 where a user's eye 245 ispositioned. For purposes of illustration, FIG. 2B shows a cross section225 associated with a single eye 245, but another optical block,separate from the optical block 230, provides altered image light toanother eye of the user.

The optical block 230 includes an electronic display element 235 of theelectronic display 115, the optics block 118, and an eye tracking unit260. The electronic display element 235 emits image light toward theoptics block 118. The optics block 118 magnifies the image light, and insome embodiments, also corrects for one or more additional opticalerrors (e.g., distortion, astigmatism, etc.). The optics block 118directs the image light to the exit pupil 250 for presentation to theuser.

The VR headset 200 includes an eye tracking unit 260 (e.g., the eyetracking unit of the eye tracking system 160 of FIG. 1). The eyetracking unit 260 includes illumination sources and optical sensors. Inone embodiment, the eye tracking unit 260, as shown in FIG. 2B, includestwo illumination sources 262 and 264, and an optical sensor 266 for eacheye. The illumination sources and the optical sensor of the eye trackingunit 260 are coupled to a control module (not shown in FIG. 2B) thatperforms the necessary data processing for generating the eye model. Thecontrol module is located within the VR headset 105 and/or the VRconsole 110. Also, in some embodiments, there is at least one eyetracking unit 260 for the left eye of the user and at least one eyetracking unit 260 for the right eye of the user.

The illumination sources 262 and 264 and optical sensor 266 are used forcorneal sphere tracking of a user's eye. The eye tracking unit 260 ispositioned within the optical block 230 such that the optical sensor 266(e.g., camera) can capture images of the user's eye (and specificallycornea of the eye) over a range of eye motion. The illumination sources262 and 264 emit light such that when the emitted light reflects off ofthe user's eye while the user views the emitted light, the opticalsensor 266 captures one or more images of the user's eye. The eyetracking unit 260 is positioned within the optical block 230 such thatlight emitted from the illumination sources 262 and 264 reaches theuser's eye through the optics block 118. The eye tracking unit 260 maybe positioned either on-axis along the user's vision (e.g., as shown inFIG. 2B) or can be placed off-axis from the user's vision (e.g., to theleft of the optics block 118). An example of corneal sphere trackingsystem is described further below with reference to FIG. 3.

Eye Tracking System for Generating Eye Model

FIG. 3 depicts an example eye tracking system 300, in accordance with anembodiment. In some embodiments, the eye tracking system 300 is part ofthe eye tracking system 160 in the VR headset 105. In alternateembodiments, the eye tacking system 300 is part of some other device,e.g., a heads up display in an AR system, or some other system utilizingeye tracking. The eye tracking system 300 includes, among othercomponents, an eye tracking unit 360 and a control module 370. Forsimplification, the discussion of the eye tracking system 300 is withregard to a single eye of the user. However, in some embodiments,corresponding eye tracking units 360 may be employed for each of theuser's eyes. In such embodiments, a single control module 370 maycontrol the multiple eye tracking units 360.

The eye tracking unit 360 includes, among other components, two or moreillumination sources (e.g., illumination sources 362 and 364) and one ormore optical sensors (e.g., optical sensor 366). The illuminationsources (e.g., point light sources) and optical sensors (e.g., camera)of the eye tracking unit are used for corneal sphere tracking and todetermine a model of an eye of a user while the user is wearing the VRheadset 105. The illumination sources 362 and 364 have well-knownemission characteristics such as ideal point light sources. In oneembodiment, two illumination sources are used. Alternatively, more thantwo illumination sources, such as, a ring of illumination sources areused. For example, the ring of illumination sources can be positionedeither in the same two-dimensional plane or arbitrary positions relativeto a reference point (e.g., location of an entrance pupil of the HMD orreference point 215). In one embodiment, the illumination sources can belocated outside of the user's line of sight. Illumination sourcespositioned arbitrarily from the reference point can be placed atdifferent depths from the reference point and/or at non-uniform spacingbetween the sources to improve the accuracy of the eye tracking.

In some embodiments, the two or more illumination sources comprisedifferent characteristics for either all of the illumination sources orbetween the illumination sources. For example, light originating fromthe two or more illumination sources can include one or more of:different wavelengths, modulated at different frequencies or amplitudes(i.e., varying intensity), have different temporal coherence thatdescribes the correlation between the two light waves at differentpoints in time, and multiplexed in either time or frequency domain.

The optical sensor 366 captures images of the user's eye to capturecorneal reflections (e.g., reflections from cornea of the eye). Forexample, the optical sensor 366 is a camera that can capture stillpictures or video. The optical sensor 366 has a plurality of parameterssuch as focal length, focus, frame rate, ISO, sensor temperature,shutter speed, aperture, resolution, etc. In some embodiments, theoptical sensor 366 has a high frame rate and high resolution. Theoptical sensor 366 can capture either two-dimensional images orthree-dimensional images. The optical sensor 366 is placed such that thecorneal reflections in response to the light from the illuminationsources incident upon the eye can be captured over a range of eyemovements (e.g., a maximum possible range). For example, when a ring ofillumination sources are placed around the eye, the optical sensor 366is placed pointed towards the eye around the center of the ring (e.g.,in the line of sight of the user). Alternatively, the optical sensor 366is placed off-axis such that it is outside of the main line of sight ofthe user. In one embodiment, more than one optical sensor 366 can beused per eye to capture corneal reflections of the eye while light fromillumination sources is incident upon the eye. The optical sensor 366may be a detector that can measure a direction of corneal reflectionssuch as column sensors, waveguides, and the like.

The illumination sources 362 and 364 emit light that is reflected at thecornea, which is then captured (e.g., as an image) at the optical sensor366. For example, light rays represented by arrows 362-I and 364-Ioriginating at the illumination sources 362 and 364 are incident uponthe eye. When light is incident upon the human eye, the eye producesmultiple reflections such as a reflection from the outer surface of thecornea and another reflection from the inner surface of the cornea. Inone embodiment, the optical sensor 366 captures the reflected light fromthe outer surface of the cornea in the captured images. For example,reflected light represented by arrows 362-R and 364-R is captured inimages captured by the optical sensor 366. Alternatively oradditionally, the optical sensor 366 captures the reflected light fromthe inner surface of the cornea. The reflections from the cornea (e.g.,from inner surface and/or outer surface) are herein referred to ascorneal reflections. An example process of generating an eye model usingcorneal sphere eye tracking is further described below with reference toFIG. 4.

The control module 370 generates eye models, performs calibration of theHMD system, and performs optical actions. For example, the controlmodule 370 performs the process to generate an eye model for one or bothof a user's eyes. In some embodiments, a single control module 370 maycontrol the multiple eye tracking units 360 such as one eye trackingunit 360 for the left eye and another eye tracking unit 360 for theright eye. The control module 370 is located within the VR headset 105and/or the VR console 110. The control module 370 is coupled with theeye tracking unit 360 such that the illumination sources 362 and 364,and optical sensor 366 can communicate with the control module 370.

An example process for generating an eye model, described below inconjunction with FIG. 4, includes the steps of turning on theillumination sources 362 and 364, capturing images including cornealreflections at the optical sensor 366 while the user is viewing knownlocations on the VR headset 105, and further processing of the capturedimages to generate an eye model. An example calibration process,described below in conjunction with FIG. 6, includes capturing imageswith corneal reflections and processing the captured images to determinea 3D plane where the pupil of the eye resides and a pupillary axis forthe eye. The calibration process may also include determining an angularoffset between the pupillary axis and the eye's true line of sight. Thecontrol module 370 may also perform one or more optical actions such asdetermining a user's gaze direction, a user's vergence angle (orvergence depth), a user's accommodation depth, identification of theuser, an eye's torsional state, or some combination thereof.

The eye tracking system 300 generates a model for the user's eye. In oneembodiment, the user's eye is modeled as two spheres 305 and 310 ofdifferent radii, where sphere 305 approximates the overall eye and aportion of sphere 310 approximates the cornea of the eye. The center (ororigin) of the sphere 305 is represented by point 306 and the center ofcorneal sphere 310 is represented by point 311. Element 315 representsthe lens of the eye and element 316 represents the pupil of the eye. Inother embodiments, the cornea may be modeled as a complex surface.

The human eye can be modeled using two spheres with a bigger sphere(i.e., sphere 305) representing an approximation of the overall eye anda smaller sphere (i.e., sphere 310) representing an approximation of aportion of the cornea of the eye, where the two spheres have differentradii and their centers are offset from each other. While it is knownthat the cornea forms only a small curved portion in the eye and is nota sphere in and off itself, the cornea can be approximated as a portionof a sphere. In one embodiment, the sphere 305 has a radius ofapproximately 25 mm and corneal sphere 310 has a radius of approximately8 mm. The centers of the sphere 305 and sphere 310 are offset from eachother as shown in FIG. 3. When the eye rotates while viewing content ona head mounted display (e.g., a VR headset 105), the rotation of the eyecauses a corresponding displacement of the center of the corneal sphere310 (i.e., point 311). The tracking system 300 tracks the motion of thecenter of the corneal sphere 310 during the rotation of the eye. In someembodiments, the eye tracking system 300 generates a model for a singleeye of the user as described below in conjunction with FIG. 4. Thegenerated eye model includes eye information including but not limitedto radius and origin information for the corneal sphere 310, which isreferred to herein as “eye model information.” Alternatively oradditionally, the eye model information may include radius and origininformation of the sphere that approximates the overall eye (i.e.,sphere 305). In one embodiment, the corneal motion may be modeled as arotation about the fixed center 306 of sphere 305. In other embodiments,the center 306 of sphere 305 is a function of the cornea position(modeling aspheric eye shapes, oculomotor muscle control, deformationunder rotation, etc.). The generated eye model information is stored ina database located within the system environment 100 or outside of thesystem environment 100.

The control module 370 uses the stored eye models to perform one or moreoptical actions (e.g., estimate gaze direction for one or both eyes ofthe user) in the normal mode of operation while the user is viewingcontent. The control module 370 receives eye tracking information fromthe eye tracking unit 360 along with the eye model information toperform optical actions such as determining a user's gaze direction,determining a user's vergence angle (or vergence depth), a user'saccommodation depth, identification of the user, an eye's torsionalstate, or some combination thereof. The control module 370 can beimplemented in either hardware, software, or some combination thereof.

FIG. 4 is a flowchart of an example process 400 for generating an eyemodel using an eye tracking system (e.g., eye tracking system 300 ofFIG. 3), in accordance with an embodiment. The example process 400 ofFIG. 4 may be performed by the eye tracking system 300, e.g., as part ofa VR headset 105 and/or the VR console 110, or some other system (e.g.,an AR system). Other entities may perform some or all of the steps ofthe process in other embodiments. Likewise, embodiments may includedifferent and/or additional steps, or perform the steps in differentorders. The example process of FIG. 4 is for generating an eye model forone of the user's eyes and can also be implemented (either concurrentlyor sequentially) for determining an eye model for the user's other eye.The example process is described using two illumination sources and oneoptical sensor for modeling the eye. An eye model can be generated usingmore than two illumination sources and/or more than one optical sensor.

The eye tracking system 300 illuminates 410 the user's eye by turning ontwo illumination sources (e.g., illumination sources 362 and 364) thatare positioned at known locations relative to, e.g., an optical sensor(e.g., optical sensor 366). These illumination sources emit light thatis incident on the user's eye such that the cornea of the eye reflectsthe light.

The eye tracking system 300 captures 420 corneal reflections of thelight incident upon the user's eye as one or more images. In oneembodiment, the eye tracking system 300 captures a single image of thecorneal reflections. Alternatively, the eye tracking system 300 capturesmultiple images of corneal reflections as the user looks at a sequenceof known targets (e.g., specific points on the electronic display 235).

The eye tracking system 300 generates a model of the eye, where the eyemodel includes radius and origin information for the corneal sphere 310.The captured one or more images including corneal reflections of lightoriginating from the two illuminations sources at known locationsresults in a single corneal radius and a corneal sphere origin that fitsthe data of the captured images. In one embodiment, the eye trackingsystem 300 generates the eye model by using a corneal learning modelthat uses a reference eye. The reference eye, as referred to herein, isan eye with a reference eye model that includes known radius and origininformation for each of corneal sphere (e.g., sphere 310) and the sphererepresenting the overall eye (e.g., sphere 305). The corneal learningmodel includes capturing the corneal reflections of the reference eyewhile the reference eye is rotated to emulate a range of human eyemovements. The corneal learning model includes the relationship betweenvarious movements of the reference eye and the corneal reflections ofthe reference eye, and such relationship is used to generate the eyemodel for the user's eye. For example, the corneal reflections of theuser's eye captured in step 420 are compared with that of the cornealreflections of the reference eye to extrapolate different parameters forthe eye model (e.g., radius and origin) for the corneal sphere 310 ofthe user's eye. In such example, the radius and origin information ofthe reference eye model of the corneal learning model is extrapolated toestimate radius and origin information for the user's eye model. Anexample method of estimating the radius and origin for the user's eyemodel is described below.

The eye tracking system 300 determines 430 a radius of the cornealsphere 310 based on the captured images that include corneal reflectionsof at least two of the illuminating sources. Each image providessufficient information to derive an estimate of the corneal radius andcorneal position. In some embodiments, the eye model may include only asphere of fixed radius for the cornea. The corneal radius estimate maythen be refined by combining estimates over a sequence of frames usingan appropriate statistical model. For example, the samples of cornealradius may form a normal distribution about the mean over a sequence ofimages. When a sequence of images provides a set of estimations with alow variance around the mean with few outliers, the corneal radius couldbe assumed to be the mean of these samples. In embodiments with three ormore illumination sources, the difference in corneal radius predicted byeach set of two illumination sources may be used to inform anon-spherical model for the corneal surface (ex. b-splines, polynomialsurface, etc.). In yet other embodiments, the difference in cornealradius predicted in a set of frames may be combined with a predictedgaze direction per frame to inform a more complex corneal model. Inthese systems, the cornea is defined by the reference origin 311 and anorientation.

The eye tracking system 300 determines an effective center 306 of eyerotation and a distance separating the eye rotation center and thecornea center (e.g., a distance between centers 306 and 311). In oneimplementation, the eye center 306 may be assumed to be a fixed point.In this system, the user may be presented with visual targets on theelectronic display element 235 presented at known angular offsets from areference position. For example, the target may be placed at a distanceapproaching infinity with the desired angular deviation in adistortion-corrected virtual rendered space of the HMD. Thedistortion-corrected virtual rendered space is a computer-generatedthree-dimensional representation of content on the electronic displayelement 235 to a user of the HMD, where the displayed content iscorrected for distortion-based optical errors. The eye tracking system300 determines and records 440 a location of the corneal sphere whilethe user is looking toward each of these calibration targets. Thissystem may model the user's gaze as lying along the vector from the eyecenter 306 through the corneal center 311. The position of the eyecenter 306 and orientation of the eye to the virtual rendered space ofthe HMD may be inferred as the point 306 and orientation that bestsatisfies the assumption that the distance 306 to 311 remains unchangedand that the gaze vectors from 306 to recorded corneal centers 311 alignmost closely with the presented target angular offsets. In othersystems, the precise eye center 306 may be modeled as a function ofcornea position (and/or cornea orientation in more complex models) forthe purpose of gaze estimation to model physiological deviations from aspherical eye model.

The generated eye model can be used in part to perform one or moreoptical actions such as determining a user's gaze direction, a user'svergence angle (or vergence depth), a user's accommodation depth,identification of the user, an eye's torsional state, or somecombination thereof. For example, a determination of where the user islooking at (i.e., user's gaze) is done using the eye model of the user.The user's gaze can be determined by determining a pupillary axis of theeye at a given time point, and further determining an offset anglebetween the eye's pupillary axis and its line of sight, where the lineof sight is indicated by the location of the fovea that is responsiblefor the eye's sharp central vision. The pupillary axis and the offsetangle can be determined during a calibration of the HMD system asdescribed below with reference to FIG. 5.

One-Time Single Point Calibration

FIG. 5 depicts an example one-time single point calibration for an HMDsystem (e.g., VR system 100 including the VR headset 105) performed byan eye tracking system (e.g., eye tracking system 300), in accordancewith an embodiment. The eye tracking system depicted and described inFIG. 5 is identical to the eye tracking system 300 described above inconjunction with FIG. 3. The example calibration system of FIG. 5 isdescribed for calibrating one of the user's eyes and can also calibrate(either concurrently or sequentially) the user's other eye. In someembodiments, the calibration includes determining a pupillary axis foreach eye and/or determining an angular offset between the pupillary axisand the eye's true line of sight. In some embodiments, the calibrationof the HMD system is based on a single image of the user's eye and isperformed only once. For example, the calibration is performed the veryfirst time the user puts the VR headset 105 on and the VR headset 105stores the calibration data corresponding to the user in a memory forfuture uses by the same user.

FIG. 5 shows an eye of a user, similar to as shown in FIG. 3, modeled astwo spheres 305 and 310 of different radii, where sphere 305approximates the overall eye and a portion of sphere 310 approximatesthe cornea of the eye. The center of the sphere 305 is represented bypoint 306 and the center of corneal sphere 310 is represented by point311. Element 315 represents the lens of the eye and element 316represents the pupil of the eye. The pupil is a circular opening in thecenter of the iris of the eye that allows light to strike the retina.FIG. 5 also shows a 3D plane 515 at which the pupil 316 resides and aray 520 originating from the 3D plane 515 that is perpendicular to thecorneal surface 510 of corneal sphere 310. The ray 520 represents thepupillary axis of the eye, which is determined by a calibration processdescribed below in conjunction with FIG. 6.

The control module 370 performs the calibration for the HMD system. Thecalibration includes determining a 3D plane where the pupil resides anda pupillary axis for the eye. The 3D plane described herein is a planein the three-dimensional space where the pupil resides. Thethree-dimensional space is defined relative to a reference point suchas, for example, the location of the optical sensor 366. The calibrationincludes capturing images of the eye with corneal reflections andprocessing the captured images to identify a shape of the pupil that canrange from a circle (in an on-axis viewing position of the eye) tocomplex shapes when viewing the pupil through the refractive cornea ofan eye rotated off-axis from the camera. For example, the shape of thepupil can be identified by determining a border between the pupil(typically black in color) and the iris (typically either brown, hazel,green, gray, or blue, or some combination thereof) by looking for achange in reflected intensity at one or more light wavelengthsassociated with the pupil and iris of the eye. The 3D plane isdetermined using the identified pupil shape and the eye modelinformation.

In one embodiment, the pupil's 3D plane is identified by modifying theidentified pupil shape based on a predicted amount of eye rotation froman on-axis viewing position of the eye. As discussed above inconjunction with FIG. 2B, the pupil shape is a circle in an on-axisviewing position of the eye. An example method for determining predictedamount of eye rotation is based on a pupil learning model, where thepupil shapes of the reference eye with the reference eye model (i.e.,reference eye and reference eye model described above in conjunctionwith FIG. 4) is captured while the eye is rotated to emulate a range ofhuman eye movements. By learning the relationship between variousmovements of the reference eye and the identified pupil shapes (e.g.,various kinds of ovals), a 3D plane for the user's pupil can beestimated for any pupil shape that is captured while the user is viewingcontent on the VR headset 105. In one embodiment, the identified 3Dplane is modified to take into account the differences between thereference eye model used in the pupil learning model and the obtainedeye model for the user's eye. For example, the radius and origin of thecorneal spheres of the reference eye model used in the pupil learningmodel and that of the obtained eye model of the user are compared tomodify the location of the pupil's identified 3D plane.

In some embodiments, the control module 370 determines an angular offsetbetween the determined pupillary axis (i.e., identified by ray 520) andthe user's true line of sight represented by the location of the eye'sfovea region. In FIG. 5, the location of the center of the eye's fovealcentralis region is represented with point 530 and the eye's true lineof sight is represented with the line 535 the connects the center of thefovea 530 and the pupil 316. The eye's fovea is responsible for theeye's true line of sight (also called as sharp central vision or fovealvision). The fovea is a small, central pit composed of closely packedcones in the eye and is located on the retina. There is an angularoffset between the pupillary axis 520 and the eye's true line of sightbecause the location of the fovea doesn't align with the pupillary axis520 if the pupillary axis 520 is extended back to the retina.

The pupillary axis 520 and the offset angle can be determined during thecalibration of the HMD system. In one embodiment, the user is asked tofixate on a target that is presented to the user at a known directionvia the VR headset 105's display. While the user is looking at thetarget, one or more images of the user's eye are captured to capture theshape of the pupil. As the location of the user's eye is known (e.g.,eye model information with corneal sphere radius and origin, and/or 3Dplane and pupillary axis providing location of the pupil) and thedirection of the presented target is also known, the user's line ofsight while viewing the target is determined. For example, the user'sline of sight is determined by a line that parallels the direction ofthe target and the known location of the pupil in a 3D space relative toa reference point such as the location of optical sensor 366, which isreferenced to the coordinate system of the HMD.

The control module 370 estimates a predicted amount of rotation theuser's eye would make from an on-axis position (i.e., a position of theeye that results in a circular pupil shape at the optical sensor 366 fora given position of the optical sensor 366) to a position to view thetarget on the display by using the pupil learning model and the user'seye model information (e.g., user's eye model generated as describedabove in conjunction with FIG. 4). From the captured one or more imagesof the pupil while the user is fixated on the target, a shape of thepupil is identified (e.g., an oval shape). Using the pupil learningmodel for the reference eye, the control module 370 estimates apredicted amount of rotation for the reference eye for the identifiedshape of the pupil. The control module then applies a correction factorto the estimated predicted amount of rotation to factor in thedifference between the reference eye model and the user's eye model. Thecorrected amount of predicted eye rotation is then applied to thepupillary axis 520 of the user's eye to determine a location of themodified pupillary axis that corresponds to the scenario when the useris fixated at the target. An angular delta between the modifiedpupillary axis and the user's line of sight when the user is fixated atthe target location represents an angular offset between the pupillaryaxis 520 and the eye's true line of sight. An example process forcalibrating the HMD system including determining a pupillary axis and/oran angular offset between the pupillary axis and the user's true line ofsight is described below with reference to FIG. 6.

FIG. 6 is a flowchart of an example process 600 for a one-time singlepoint calibration of an HMD system (e.g., VR system environment 100including the VR headset 105 or an AR system including an AR headset),in accordance with an embodiment. The example calibration process 600 ofFIG. 6 may be performed by the eye tracking system 300, e.g., as part ofthe VR headset 105 and/or the VR console 110 or some other system (e.g.,an AR system). Other entities may perform some or all of the steps ofthe process in other embodiments. Likewise, embodiments may includedifferent and/or additional steps, or perform the steps in differentorders. The calibration process of FIG. 6 is for calibrating the HMDsystem for one of the user's eyes and can also be implemented (eitherconcurrently or sequentially) for calibrating the HMD system for theuser's other eye. In some embodiments, the calibration process 600 isperformed only once. For example, the calibration process 600 isperformed the very first time the user puts the VR headset 105 on andthe VR headset 105 stores the calibration data corresponding to the userin a memory for use in future uses by the same user.

The eye tracking system 300 obtains 610 eye model information includingradius and origin information of corneal sphere 310 of the user's eye.In one embodiment, the eye tracking system 300 obtains the eye modelinformation from within the eye tracking system 300 (e.g., eye trackingsystem 300 generates the eye model as described above in conjunctionwith FIG. 4). Alternatively, the eye tracking system 300 obtains the eyemodel information external to the eye tracking system 300 (e.g., VRconsole 110 or outside of system environment 100).

The eye tracking system 300 illuminates 620 the user's eye by turning ontwo illumination sources (e.g., illumination sources 362 and 364) thatare positioned at known locations relative to, e.g., an optical sensor(e.g., optical sensor 366). These illumination sources emit light thatis incident on the user's eye such that the cornea of the eye reflectsthe light.

The eye tracking system 300 captures 630 one or more images of a user'scornea (i.e., corneal reflections). The images capturing the cornealreflections are very similar to those described above in conjunctionwith FIG. 4. In one embodiment, the eye tracking system 300 captures 630one or more images while the user is looking at one or more knowntargets (e.g., specific points on the electronic display 235). Forexample, a single point calibration is performed where the user onlyneeds to look at one specific target on the electronic display 235.Alternatively, the system 300 captures images when the user is viewingcontent displayed on the VR headset 105 in a normal mode of operation.

The eye tracking system 300 identifies 640 the shape of the eye's pupil316 based on the captured one or more images. When the eye is viewingcontent such that the optical sensor 366 is on-axis (i.e., along thepupillary axis 520 of the eye), the pupil shape is captured as a circlein the images captured by the optical sensor 366. When the eye isviewing content such that the optical sensor 366 is off-axis (i.e., notalong the pupillary axis 520 of the eye), the pupil shape isapproximated as an ellipsoid. In one embodiment, the pupil shape isidentified by processing a single captured image. For example, the shapeof the pupil can be identified by determining the border between thepupil and the iris by looking for a change in reflected intensity ofvarious wavelengths of light associated with the pupil and iris of theeye. The pupil is typically black in color and the iris is typically oneof brown, hazel, green, gray, or blue. As the pupil is an opening withinthe iris region, a shape of the pupil can be identified by observing aborder between the black pupil and a different reflected intensity atone or more light wavelengths associated with the iris region.Alternatively, the pupil shape is identified by processing the diffuseillumination corresponding to a plurality of captured images.

In one embodiment, the identified pupil shape is modified to correct forrefractive distortion of the cornea. For example, the correction for thecornea's refractive distortion can initially be implemented based on atypical human eye (e.g., based on empirical data) and can later belearned more accurately for the particular eye of the user. In someembodiments, the cornea may be approximated as a sphere with a typicalindex of refraction for human corneas (e.g., 1.377). The pupillary 3Dplane 515 may be assumed to be located at a typical distance behind thecorneal surface 510 (e.g., 3 mm). The radius of the corneal sphere 310and distance to the corneal surface 310 may be known via cornealtracking described above in conjunction with FIGS. 3 and 4. For a givenpoint in the image plane determined to be on the edge of the pupil 316,a ray may be found that passes through this point when refracted by thecornea of known shape, distance, and assumed index of refraction. Therefracted ray then travels an additionally assumed distance to strikethe pupil 3D plane 515 at the location of the actual pupil border. Inother implementations, the corneal index of refraction and distance topupil plane may be measured during calibration or learned over course ofuse.

The eye tracking system 300 identifies 650 a 3D plane 515 that the pupilresides in by first correcting the pupil edge shape aberrations in imagesensor space caused by camera distortion and corneal refraction, thenfinding the best projection onto a plane to yield an at-rest pupil shape(ex. a circle).

The eye tracking system 300 determines 660 a pupillary axis (e.g.,pupillary axis 520) for the eye, where the pupillary axis 520 is derivedby identifying a ray originating from the pupil's 3D plane and isperpendicular to a surface of the corneal sphere 310 of the eye. Becausethe pupil's 3D plane (e.g., plane represented by line 515) is offsetfrom the center of the corneal sphere (e.g., point 311), there is asingle ray that is originating from the pupil's 3D plane and isperpendicular to a surface of the corneal sphere 310 of the eye.

In one embodiment, the determined pupillary axis 520 is used todetermine an angular offset between the pupillary axis 520 and the eye'strue line of sight that is represented by the position of the center 530of the eye's foveal centralis region. One example method of determiningthe angular offset is to ask the user to fixate on a target that islocated at a known position on the VR headset 105's display, and thenmeasure the angular delta between the pupillary axis (i.e., pupillaryaxis 520 adjusted for a predicted amount of rotation the user's eyeundergoes when fixating on the presented target) and the line of sightto the presented target. The measured angular delta represents anangular offset between the pupillary axis 520 and the eye's true line ofsight.

The eye tracking system 300 performs 670 one or more optical actionsbased in part on the determined pupillary axis 520 and/or the angularoffset. Example optical actions include determining a user's gazedirection, a user's vergence angle (or vergence depth), a user'saccommodation depth, identification of the user, and eye's torsionalstate, or some combination thereof. For example, a determination ofwhere the user is looking at (i.e., user's gaze) can be made bycapturing an image of the pupil while the user is viewing in aparticular direction. As described above, the eye tracking system 300can compare the captured pupil image to that of the images of the pupillearning model and using the obtained eye model information can estimatethe user's gaze. Other example actions such as identification of theuser and eye's torsional state can be determined by tracking the iris ofthe eye. For example, a user's iris (e.g., for one or more eyes) can becaptured in an image and can be stored in memory (e.g., at VR headset105, VR console 110, or on the cloud). The stored iris image can laterbe used for identifying the user by comparing it with a later capturediris image. Some of the example optical actions might require capturedcorneal reflection data and/or calibration data (e.g., pupillary axisand/or angular offset to true light of sight) of both the user's eyes toperform the optical actions, and one or more steps of the exampleprocesses 400 and/or 600 may be implemented (either concurrently orsequentially) with respect to the user's other eye to perform suchoptical actions.

Additional Configuration Information

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the patent rights be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A headset comprising: an eye tracking systemincluding two or more illumination sources configured to illuminate aneye of a user, the eye tracking system configured to: obtain a model ofthe eye of the user including a radius of a corneal sphere of the eyeand location of a center of the corneal sphere; capture one or moreimages of reflections of the eye of the user reflected from the eye ofthe user; determine, from the one or more images of the reflections, ashape of a pupil of the eye of the user; determine a plane correspondingto a location of the pupil, the plane being identified using thedetermined shape of the pupil, the radius of the corneal sphere, and thelocation of the center of the corneal sphere from the model of the eye;determine, from the plane, a pupillary axis for the eye based on theidentified plane parallel to the pupil; and perform an optical actionbased on the determined pupillary axis for the eye.
 2. The headset ofclaim 1, wherein the model of the eye is generated by: capturing animage of one or more reflections of the eye reflected from a cornea ofthe eye of the user; determining a radius of a corneal sphere of thecornea based on the captured image of the reflections; and determining acenter of the corneal sphere based on the determined radius of thecorneal sphere.
 3. The headset of claim 1, wherein the optical action isselected from the group comprising: determining a gaze direction of theuser, determining a vergence angle of the eye of the user, determiningthe vergence depth of the user, determining an accommodation depth forthe user, performing user identification, determining a torsional stateof the eye of the user, or some combination thereof.
 4. The headset ofclaim 1, wherein determining the shape of the pupil based on the one ormore captured pupil reflections includes: determining, from the one ormore reflections, a border between the pupil and an iris of the eyebased on a change in reflected intensity of one or more wavelengths oflight; and identifying the shape of the pupil from the determinedborder.
 5. The headset of claim 1, wherein the user is viewing contentdisplayed by the headset while the one or more images of reflections ofthe eye of the user reflected from the cornea of the eye of the user arereflected, and wherein the content includes a known calibration targetdisplayed on an electronic display of the headset.
 6. The headset ofclaim 1, wherein the pupillary axis derived by: identifying a rayoriginating from the plane that is perpendicular to a surface of thecorneal sphere.
 7. The headset of claim 6, further comprising:identifying a foveal centralis region of the eye of the user;determining a second ray originating from the foveal centralis region ofthe eye and passing through the pupil corresponding to a true line ofsight of the user; and determining an angular offset between thedetermined pupillary axis for the pupil of the user and the true line ofsight of the user.
 8. A method comprising: capturing, by an eye trackingsystem of a head mounted display (HMD), one or more images ofreflections of an eye of a user, the eye tracking system including twoor more illumination sources configured to illuminate the eye of theuser; identifying, from the one or more images of the reflections, ashape of a pupil of the eye of the user; identifying a plane parallel tothe pupil based on the identified shape of the pupil determining apupillary axis for the eye based on the identified plane parallel to thepupil; and performing an optical action based on the determinedpupillary axis.
 9. The method of claim 8, wherein the plane parallel tothe pupil is determined based on the shape of the pupil and furtherbased on a model of the eye of the user, wherein the model of the eye isgenerated by: capturing an image of one or more reflections of the eyereflected from a cornea of the eye of the user; determining a radius ofa corneal sphere of the cornea based on the captured image of thereflections; and determining a center of the corneal sphere based on thedetermined radius of the corneal sphere.
 10. The method of claim 8,wherein identifying the shape of the pupil comprises: determining, fromthe one or more captured pupil reflections, a border between the pupiland an iris of the eye based on a change in reflected intensity of oneor more wavelengths of light; and identifying the shape of the pupilfrom the determined border.
 11. The method of claim 8, wherein theoptical action is selected from the group comprising: determining a gazedirection of the user, determining a vergence angle of the eye of theuser, determining the vergence depth of the user, determining anaccommodation depth for the user, performing user identification,determining a torsional state of the eye of the user, or somecombination thereof.
 12. The method of claim 8, wherein the user isviewing content displayed by the HMD while the one or more images ofreflections of the eye of the user reflected from the cornea of the eyeof the user are reflected, and wherein the content includes a knowncalibration target displayed on an electronic display of the HMD. 13.The method of claim 8, wherein the pupillary axis derived by:identifying a ray originating from the plane that is perpendicular to asurface of a corneal sphere of the eye of the user.
 14. The method ofclaim 13, further comprising: identifying a foveal centralis region ofthe eye of the user; determining a second ray originating from thefoveal centralis region of the eye and passing through the pupilcorresponding to a true line of sight of the user; and determining anangular offset between the determined pupillary axis for the pupil ofthe user and the true line of sight of the user.
 15. A headsetcomprising: an eye tracking system including two or more illuminationsources configured to illuminate an eye of a user, the eye trackingsystem configured to: capture one or more images of reflections of theeye of the user reflected from a cornea of the eye of the user;determine, from the one or more images of the reflections, a shape of apupil of the eye of the user; determine a plane corresponding to alocation of the pupil, the plane being identified using the determinedshape of the pupil and a model of the eye of the user; determine, fromthe determined shape of the pupil and the model of the eye of the user,a center of the corneal sphere; and perform an optical action based inpart on the determined center of the corneal sphere.
 16. The headset ofclaim 15, wherein the model of the eye is generated by: capturing animage of one or more reflections of the eye reflected from a cornea ofthe eye of the user; determining a radius of a corneal sphere of thecornea based on the captured image of the reflections; and determining acenter of the corneal sphere based on the determined radius of thecorneal sphere.
 17. The headset of claim 15, wherein the optical actionis selected from the group comprising: determining a gaze direction ofthe user, determining a vergence angle of the eye of the user,determining the vergence depth of the user, determining an accommodationdepth for the user, performing user identification, determining atorsional state of the eye of the user, or some combination thereof. 18.The headset of claim 15, wherein determining the shape of the pupilbased on the one or more captured pupil reflections includes:determining, from the one or more captured pupil reflections, a borderbetween the pupil and an iris of the eye based on a change in reflectedintensity of one or more wavelengths of light; and identifying the shapeof the pupil from the determined border.
 19. The headset of claim 15,wherein the user is viewing content displayed by the headset while theone or more images of reflections of the eye of the user reflected fromthe cornea of the eye of the user are reflected, and wherein the contentincludes a known calibration target displayed on an electronic displayof the headset.
 20. The headset of claim 15, wherein the pupillary axisderived by: identifying a ray originating from the plane that isperpendicular to a surface of the corneal sphere.